• Sentiment =

  • feelings – Attitudes – Emotions – Opinions

  • A thought, view, or attitude, especially one based mainly on emotion instead of reason

• Subjective impressions, not facts

• Emotion: brief organically synchronized … evaluation of a major event
  • angry, sad, joyful, fearful, ashamed, proud, elated
• Mood: diffuse non-caused low-intensity long-duration change in subjective feeling
  • cheerful, gloomy, irritable, listless, depressed, buoyant
• Interpersonal stances: affective stance toward another person in a specific interaction
  • friendly, flirtatious, distant, cold, warm, supportive, contemptuous
• Attitudes: enduring, affectively colored beliefs, dispositions towards objects or persons
  • liking, loving, hating, valuing, desiring
• Personality traits: stable personality dispositions and typical behavior tendencies
  • nervous, anxious, reckless, morose, hostile, jealous

• Sentiment Analysis

  • use of natural language processing (NLP) and computational techniques to automate the extraction or classification of sentiment from typically unstructured text

• Opinion mining

• Sentiment mining

• Subjectivity analysis

• Book: is this review positive or negative?

• Products: what do people think about the new iPhone?

• Blog:

• Politics: what do people think about this candidate or issue?

• Twitter:

• Movie: is this review positive or negative?

• Marketing: how is consumer confidence? Consumer attitudes? Trend?

• Prediction: predict election outcomes or market trends from sentiment

• Healthcare:

• Regular opinions: Sentiment/opinion expressions on some target entities

  • Direct opinions:

    • “The touch screen is really cool.”

  • <span style="color:blue>Indirect opinions:

    • “After taking the drug, my pain has gone.”

• Comparative opinions: Comparison of more than one entity.

  • E.g., “iPhone is better than Blackberry.”

• We focus on regular opinions first, and just call them opinions.

• An opinion is a quintuple
  
  ( entity, aspect, sentiment, holder, time)
  
  where

  • entity: target entity (or object).

  • Aspect: aspect (or feature) of the entity.

  • Sentiment: +, -, or neu, a rating, or an emotion.

  • holder: opinion holder.

  • time: time when the opinion was expressed.

• Id: Abc123 on 5-1-2008 “I bought an iPhone a few days ago. It is such a nice phone. The touch screen is really cool. The voice quality is great too. It is much better than my old Blackberry, which was a terrible phone and so difficult to type with its tiny keys. However, my mother was mad with me as I did not tell her before I bought the phone. She also thought the phone was too expensive”

• Simplest task:

  • Is the attitude of this text positive or negative?

• More complex:

  • Rank the attitude of this text from 1 to 5

• Advanced:

  • Detect the target, source, or complex attitude types

• Which features to use?

  • Words (unigrams)
  • Phrases/n-grams
  • Sentences

• How to interpret features for sentiment detection?

  • Bag of words (IR)
  • Annotated lexicons (WordNet, SentiWordNet)
  • Syntactic patterns
  • Paragraph structure

• Harder than topical classification, with which bag of words features perform well

• Must consider other features due to…

  • Subtlety of sentiment expression
    • irony
    • expression of sentiment using neutral words
  • Domain/context dependence
    • words/phrases can mean different things in different contexts and domains
  • Effect of syntax on semantics

• Lexicon-based methods (dictionary-based)

  • Using sentiment words and phrases: good, wonderful, awesome, troublesome, cost an arm and leg

  • Not completely unsupervised!

• Supervised learning methods: to classify reviews into positive and negative.
  • Machine learning
    • Naïve Bayes, Maximum Entropy, Support Vector Machines (SVM)
  • Recent research
    • Deep learning

• Home page: http://www.wjh.harvard.edu/~inquirer

• List of Categories: http://www.wjh.harvard.edu/~inquirer/homecat.htm

• Spreadsheet: http://www.wjh.harvard.edu/~inquirer/inquirerbasic.xls

  • Categories:
    • Positiv (1915 words) and Negativ (2291 words)
    • Strong vs Weak, Active vs Passive, Overstated versus Understated
    • Pleasure, Pain, Virtue, Vice, Motivation, Cognitive Orientation, etc

• Free for Research Use
  
  
  
  Philip J. Stone, Dexter C Dunphy, Marshall S. Smith, Daniel M. Ogilvie. 1966. The General Inquirer: A Computer Approach to Content Analysis. MIT Press

• Home page: http://www.liwc.net/

• 2300 words, >70 classes

• Affective Processes

  • negative emotion (bad, weird, hate, problem, tough)
  • positive emotion (love, nice, sweet)

• Cognitive Processes

  • Tentative (maybe, perhaps, guess), Inhibition (block, constraint)

• Pronouns, Negation (no, never), Quantifiers (few, many)

Pennebaker, J.W., Booth, R.J., & Francis, M.E. (2007). Linguistic Inquiry and Word Count: LIWC 2007. Austin, TX

• Home page: http://www.cs.pitt.edu/mpqa/subj_lexicon.html

• 6885 words from 8221 lemmas

  • 2718 positive
  • 4912 negative

• Each word annotated for intensity (strong, weak)

• GNU GPL

Theresa Wilson, Janyce Wiebe, and Paul Hoffmann (2005). Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis. Proc. of HLT-EMNLP-2005.
Riloff and Wiebe (2003). Learning extraction patterns for subjective expressions. EMNLP-2003.

• Bing Liu’s Page on Opinion Mining

• http://www.cs.uic.edu/~liub/FBS/opinion-lexicon-English.rar

• 6786 words

  • 2006 positive
  • 4783 negative

Minqing Hu and Bing Liu. Mining and Summarizing Customer Reviews. ACM SIGKDD-2004.

• Home page: http://sentiwordnet.isti.cnr.it/

• All WordNet synsets automatically annotated for degrees of positivity, negativity, and neutrality/objectiveness

• [estimable(J,3)] “may be computed or estimated”
  \[\operatorname{Pos\ \ 0\ \ \ Neg\ \ 0\ \ \ Obj\ \ 1} \]

• [estimable(J,1)] “deserving of respect or high regard” \[\operatorname{Pos\ \ .75\ \ \ Neg\ \ 0\ \ \ Obj\ \ .25} \]
  
  
  Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010 SENTIWORDNET 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining. LREC-2010

• Is logical negation (no, not) associated with negative sentiment?

• Potts experiment:

  • Count negation (not, n’t, no, never) in online reviews
  • Regress against the review rating

• Use a small amount of information
  • A few labeled examples
  • A few hand-built patterns
• To bootstrap a lexicon

• Adjectives conjoined by “and” have same polarity
  • Fair and legitimate, corrupt and brutal
  • *fair and brutal, *corrupt and legitimate
• Adjectives conjoined by “but” do not
  • fair but brutal

• Label seed set of 1336 adjectives (all >20 in 21 million word WSJ corpus)
  • 657 positive
    • adequate central clever famous intelligent remarkable reputed sensitive slender thriving…
  • 679 negative
    • contagious drunken ignorant lanky listless primitive strident troublesome unresolved unsuspecting…

• Expand seed set to conjoined adjectives

• Supervised classifier assigns “polarity similarity” to each word pair, resulting in graph:

• Clustering for partitioning the graph into two

• Positive
  • bold decisive disturbing generous good honest important large mature patient peaceful positive proud sound stimulating straightforward strange talented vigorous witty…
• Negative
  • ambiguous cautious cynical evasive harmful hypocritical inefficient insecure irrational irresponsible minor outspoken pleasant reckless risky selfish tedious unsupported vulnerable wasteful…

1. Extract a phrasal lexicon from reviews
2. Learn polarity of each phrase
3. Rate a review by the average polarity of its phrases

• Positive phrases co-occur more with “excellent”

• Negative phrases co-occur more with “poor”

• But how to measure co-occurrence?

• Mutual information between 2 random variables X and Y

\[I(X,Y) = \sum_X \sum_Y{P(x,y)log_2{\frac{P(x,y)}{P(x)P(y)}}}\]

• Pointwise mutual information:
  • How much more do events x and y co-occur than if they were independent?

\[PMI(X,Y)=log_2{\frac{P(x,y)}{P(x)P(y)}}\]

• Pointwise mutual information:
  • How much more do events x and y co-occur than if they were independent?

\[PMI(X,Y)=log_2{\frac{P(x,y)}{P(x)P(y)}}\]

• PMI between two words:
  • How much more do two words co-occur than if they were independent?

\[PMI(word_1,woprd_2)=log_2{\frac{P(word_1,word_2)}{P(word_1)P(word_2)}}\]

• Query search engine (Altavista)
  • P(word) estimated by      hits(word)/N
  • P(word1,word2) by     hits(word1 NEAR word2)/N^2

\[PMI(word_1,woprd_2)=log_2{\frac{hits(word_1 \: \mathrm{NEAR} \: word_2)}{hits(word_1)hits(word_2)}}\]

\[ \begin{align} \mathrm{Polarity}(phrase) = \mathrm{PMI}(pharse, \mathrm{"excellent"}) - \mathrm{PMI}(pharse, \mathrm{"poor"}) \\ \\ = log_2{\frac{hits(phrase \: \mathrm{NEAR} \: \mathrm{"excellent"})}{hits(phrase)hits(\mathrm{"excellent"})}} - log_2{\frac{hits(phrase \: \mathrm{NEAR} \: \mathrm{"poor"})}{hits(phrase)hits(\mathrm{"poor"})}} \\ \\ = log_2{\frac{hits(phrase \: \mathrm{NEAR} \: \mathrm{"excellent"})}{hits(phrase)hits(\mathrm{"excellent"})}} {\frac{hits(phrase)hits(\mathrm{"poor"})}{hits(phrase \: \mathrm{NEAR} \: \mathrm{"poor"})}} \\ \\ = log_2{(\frac{hits(phrase \: \mathrm{NEAR} \: \mathrm{"excellent"}) hits(\mathrm{"poor"})}{hits(phrase \: \mathrm{NEAR} \: \mathrm{"poor"}) hits(\mathrm{"excellent"})})} \end{align} \]

• 410 reviews from Epinions
  • 170 (41%) negative
  • 240 (59%) positive
• Majority class baseline: 59%
• Turney algorithm: 74%
  
  
• Phrases rather than words
• Learns domain-specific information

• WordNet: online thesaurus (covered in later lecture).
• Create positive (“good”) and negative seed-words (“terrible”)
• Find Synonyms and Antonyms
  • Positive Set: Add synonyms of positive words (“well”) and antonyms of negative words
  • Negative Set: Add synonyms of negative words (“awful”) and antonyms of positive words (”evil”)
• Repeat, following chains of synonyms
• Filter

• Advantages:
  • Can be domain-specific
  • Can be more robust (more words)
• Intuition
  • Start with a seed set of words (‘good’, ‘poor’)
  • Find other words that have similar polarity:
    • Using “and” and “but”
    • Using words that occur nearby in the same document
    • Using WordNet synonyms and antonyms

• Classify a whole opinion document (e.g., a review) based on the overall sentiment of the opinion holder (Pang et al 2002; Turney 2002)
  • Classes: Positive, negative (possibly neutral)
• An example review:
  • “I bought an iPhone a few days ago. It is such a nice phone, although a little large. The touch screen is cool. The voice quality is great too. I simply love it!”
  • Classification: positive or negative?
• It is basically a text classification problem

• Classify the sentiment expressed in a sentence
  • Classes: positive, negative, neutral
  • Neutral means no sentiment expressed
    • “I believe he went home yesterday.”
    • “I bought a iPhone yesterday”
• But bear in mind
  • Explicit opinion: “I like this car.”
  • Fact-implied opinion: “I bought this car yesterday and it broke today.”
  • Mixed opinion: “Apple is doing well in this poor economy”

• The problem has been studied by numerous researchers.

• Key: feature engineering. A large set of features have been tried by researchers. E.g.,

  • Terms frequency and different IR weighting schemes
  • Part of speech (POS) tags
  • Opinion words and phrases
  • Negations
  • Syntactic dependency

• Machine learning

  • Naïve Bayes (Assume pairwise independent features)

  • Maximum Entropy Classifier (Assume pairwise independent features)

  • SVM

  • Markov Blanket Classifier

    • Accounts for conditional feature dependencies
    • Allowed reduction of discriminating features from thousands of words to about 20 (movie review domain)

• Polarity detection:
  • Is an IMDB movie review positive or negative?
• Data: Polarity Data 2.0:
  • http://www.cs.cornell.edu/people/pabo/movie-review-data

• Tokenization

• Feature Extraction

• Classification using different classifiers

  • Naïve Bayes
  • MaxEnt
  • SVM

• How to handle negation

  • I didn’t like this movie
    vs
    
  • I really like this movie

• Which words to use?

  • Only adjectives

  • All words

    • All words turns out to work better, at least on this data

Add NOT_ to every word between negation and following punctuation:

\[C_{NB} = \underset{c_j \in C}{\operatorname{argmax}}P(c_j) \prod_{i \in positions}{P(w_i|c_i)} \]

\[\hat{P}(w|c) = \frac{count(w,c) + 1}{count(c) + |V|}\]

• Negation is important

• Using all words (in naïve bayes) works well for some tasks

• Finding subsets of words may help in other tasks

  • Hand-built polarity lexicons
  • Use seeds and semi-supervised learning to induce lexicons

• Explicit aspects: Aspects explicitly mentioned as nouns or noun phrases in a sentence

  • “The picture quality is of this phone is great.”

• Implicit aspects: Aspects not explicitly mentioned in a sentence but are implied

  • “This car is so expensive.”
  • “This phone will not easily fit in a pocket.”
  • “Included 16MB is stingy.”

• Some work has been done (Su et al. 2009; Hai et al 2011)

• “ Trying out Chrome because Firefox keeps crashing.”

  • Firefox - negative; no opinion about chrome.

  • We need to segment the sentence into clauses to decide that “crashing” only applies to Firefox(?).

• But how about these

  • “ I changed to Audi because BMW is so expensive.”

  • “ I did not buy BWM because of the high price.”

  • “ I am so happy that my iPhone is nothing like my old ugly Droid.”

• These two sentences are from paint reviews.

  • “ For paintX, one coat can cover the wood color.”

  • “ For paintY, we need three coats to cover the wood color

  • We know that paintX is good and paintY is not, but how, by a system.

• “My goal is to get a tv with good picture quality”

• “The top of the picture was brighter than the bottom.”

• “When I first got the airbed a couple of weeks ago it was wonderful as all new things are, however as the weeks progressed I liked it less and less.”

• Conditional sentences are hard to deal with (Narayanan et al. 2009)

  • “ If I can find a good camera, I will buy it.”

  • But conditional sentences can have opinions

    • “ If you are looking for a good phone, buy Nokia”

• Questions are also hard to handle

  • “ Are there any great perks for employees?”

  • “ Any idea how to fix this lousy Sony camera?”

• Sarcastic sentences

  • “ What a great car, it stopped working in the second day.”

• Sarcastic sentences are common in political blogs, comments and discussions.

  • They make political opinions difficult to handle

• Sentiment: Positive, Negative, Neutral

• Emotion: angry, sad, joyful, fearful, ashamed, proud, elated

• Disease: Healthy, Cold, Flu

• Weather: Sunny, Cloudy, Rain, Snow

• While some classification algorithms naturally permit the use of more than two classes and/or labels, others are by nature binary algorithms; these can, however, be turned into multinomial classifiers by a variety of strategies.

• A common strategy is one-vs-all, which involves training a single classifier per class, with the samples of that class as positive samples and all other samples as negatives.

• Train a logistic regression classifier \(h_\theta^{(i)}(x)\) for each class \(i\) to predict the probability that \(y=i\)

• Given a new input \(x\), pick the class \(i\) that maximizes

\[\max_i{h_\theta^{(i)}(x)}\]

• Multiclass classification is the task of classifying instances into one of three or more classes. Classifying instances into one of two classes is called binary classification. Multiclass classification should not be confused with multi-label classification, where multiple labels are to be predicted for each instance.

• In multiclass, one-vs-all requires the base classifiers to produce a real-valued score for its decision, rather than just a class label. Then, the final label is the one corresponding to the class with the highest score.

• In multilabel, this strategy predicts all labels for this sample for which the respective classifiers predict a positive result.

• Sentiment Analysis

• Multiclass and Multi-label classification